COMPUTER-READABLE RECORDING MEDIUM STORING RESOLUTION 
CONVERTING PROGRAM, RESOLUTION CONVERTING DEVICE AND 
RESOLUTION CONVERTING METHOD 

RELATED APPLICATIONS 

[0001] This application is based on application no. 2000-368842 filed in Japan, the 
contents of which are hereby incorporated by reference. 

FIELD OF THE INVENTION 

[0002] The present invention relates to a computer-readable recording medium 
storing a resolution converting program, a resolution converting device and a resolution 
converting method, and more particularly to a computer-readable recording medium 
storing a resolution converting program that converts resolution based on a model 
generated through statistical processing of multiple images, as well as a resolution 
converting device and a resolution converting method. 

DESCRIPTION OF THE RELATED ART 

[0003] In the conventional art, when image data is transmitted from one device to 
another, if the resolution of images that can be processed by the latter device is different 
from the resolution of images generated by the former device, it is necessary to convert 
the resolution. In such a case, the resolution is converted by performing interpolation 
such as linear or tertiary interpolation regarding the image data. 

[0004] Japanese Laid-Open Patent Application Hei 7-152907 discloses, as another 
method of resolution conversion, a method in which the wavelet component of the input 
image is predicted, and compensation is made for the high frequency component of an 





enlarged image with a higher resolution. 

[0005] However, where interpolation such as linear or tertiary interpolation is 
performed regarding an image, because only the resolution of the image before 
enlargement or the resolution of the image that has undergone smoothing but not 
enlargement is available, the problem exists that the contours of the enlarged image are 
blurry or exhibit noticeable roughness. 

[0006] According to the method disclosed by Japanese Laid-Open Patent 
Application Hei 7-152907, it is necessary to predict the wavelet component for a higher 
resolution image. When predicting the wavelet component, the results of a prescribed 
learning process must be used, giving rise to the problem that the results of the learning 
process affect image quality. 



[0007] The present invention was created in order to resolve these problems, and an 
aspect thereof is to provide a computer-readable recording medium storing a resolution 
converting program, a resolution converting device and a resolution converting method 
that can improve the quality of the resolution-converted image. 
[0008] Another aspect of the present invention is to provide a computer-readable 
recording medium storing a resolution converting program, a resolution converting 
device and a resolution converting method that can easily convert the resolution of an 
image. 

[0009] According to one embodiment of the present invention, the present invention 
comprises a computer-readable recording medium storing a resolution converting 
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program that converts resolution based on a model generated through statistical 
processing of multiple images, wherein the program is implemented using a computer 
and includes: a step for preparing multiple models having different resolutions; a step 
for receiving an input image; a step for detecting the resolution of the input image; a 
step for projecting the input image onto a model selected in accordance with the 
detected resolution in order to obtain characteristic parameters of the input image; and a 
step for generating an image by applying the obtained characteristic parameters to a 
model different from the model used to obtain the characteristic parameters. 
[0010] According to this embodiment, multiple models having different resolutions 
are prepared. The input image is projected onto a model corresponding to the resolution 
of the input image, and the characteristic parameters of the input image are obtained. 
Using the characteristic parameters thereby obtained, an image is generated using a 
model different from the model used to obtain the characteristic parameters. 
Consequently, a computer-readable recording medium storing a resolution converting 
program capable of improving the quality of the resolution-converted image can be 
provided. 

[0011] Preferably, the step for preparing multiple models having different 
resolutions has a step for converting the resolution of a model generated through 
statistical processing. 

[0012] According to this embodiment, the multiple models having different 
resolutions are generated by converting the resolution of a model generated through 
statistical processing. Among the models having different resolutions, the images 
generated based on the same characteristic parameters differ from each other only in 



terms of resolution. Therefore, because multiple resolution-converted models may be 
obtained from one model, a computer-readable recording medium storing a resolution 
converting program that can easily change the resolution of an image can be provided. 
[0013] Preferably, the step for preparing multiple models having different 
resolutions has a step for converting the resolutions of multiple images and generating a 
model by performing statistical processing regarding each image having a different 
resolution. 

[0014] According to this embodiment, the multiple models having different 
resolutions are generated by converting the resolution of each of the multiple images 
and performing statistical processing regarding each image having a different resolution. 
Therefore, because multiple models having different resolutions and generated from 
multiple images are used, a computer-readable recording medium storing a resolution 
converting program that can easily change the resolution of an image can be provided. 
[0015] Preferably, the image generation step generates an image using a model 
having a higher resolution than the detected resolution. 

[0016] According to this invention, because an image is generated based on a model 

having a higher resolution than the detected resolution, the quality of the image can be 

improved when the resolution thereof is increased. 

[0017] Preferably, the multiple images are images of a person's face. 

[0018] According to this invention, because the multiple images are images of a 

person's face, the quality of the resolution-converted image of a face can be improved. 

[0019] Preferably, a step for extracting the face area from the input image, a step for 

converting the resolution of the input image, and a step for synthesizing the resolution- 



converted image and the image generated in the image generation step are further 
included. 

[0020] According to this invention, the face area is extracted from the input image 
and the resolution of the input image is converted. The resolution-changed image and 
the image generated through the image generation step are then synthesized. Therefore, 
where a person's face is included in the input image, the quality of the image of the 
person's face may be improved. 

[0021] According to another aspect of the present invention, the resolution 
converting device includes means for preparing multiple models having different 
resolutions; means for receiving an input image; means for detecting the resolution of 
said input image; means for projecting said input image onto a model selected in 
accordance with said detected resolution in order to obtain characteristic parameters of 
said input image; and means for generating an image by applying said obtained 
characteristic parameters to a model different from said model used to obtain the 
characteristic parameters. 

[0022] According to this invention, multiple models having different resolutions are 
prepared. The input image is projected onto a model corresponding to the resolution of 
the input image, and the characteristic parameters of the input image are obtained. Using 
the thus obtained characteristic parameters, an image is generated based on a model 
different from the model used to obtain the characteristic parameters. Therefore, a 
resolution converting device that can improve the quality of the resolution-converted 
image can be provided. 

[0023] According to yet another aspect of the present invention, the present 



invention comprises a resolution converting method that converts resolution using a 
model generated through statistical processing of multiple images and includes a step 
for preparing multiple models having different resolutions; a step for receiving an input 
image; a step for detecting the resolution of said input image; a step for projecting said 
input image onto a model selected in accordance with said detected resolution in order 
to obtain characteristic parameters of said input image; and a step for generating an 
image by applying said obtained characteristic parameters to a model different from said 
model used to obtain the characteristic parameters. 

[0024] According to this invention, multiple models having different resolutions are 
prepared. The input image is projected onto a model corresponding to the resolution of 
the input image, and the characteristic parameters of the input image are obtained. Using 
the thus obtained characteristic parameters, an image is generated based on a model 
different from the model used to obtain the characteristic parameters. Therefore, a 
resolution converting method that can improve the quality of the resolution-converted 
image can be provided. 



BRIEF DESCRIPTION OF THE DRAWINGS 

[0025] Fig. 1 is a block diagram showing the hardware construction of the 
resolution converting device comprising one embodiment; 

[0026] Fig. 2 is a flow chart showing the outline of the model building feature of the 
resolution converting device pertaining to the embodiment; 

[0027] Fig. 3 is a flow chart showing the sequence of the model building process 
performed by the resolution converting device pertaining to the embodiment; 



[0028] Fig. 4 is a drawing showing a Gaussian pyramid in a summary fashion; 
[0029] Fig. 5 is a functional block diagram showing the outline of the resolution 
conversion feature of the resolution converting device pertaining to the embodiment; 
[0030] Fig. 6 is a drawing showing in a summary fashion the resolution conversion 
process performed by the resolution converting device pertaining to the embodiment; 
and 

[0031] Fig. 7 is a flow chart showing the sequence of the resolution conversion 
process performed by the resolution converting device pertaining to the embodiment. 

DESCRIPTION OF THE PREFERRED EMBODIMENTS 

[0032] An embodiment of the present invention is explained below with reference to 
the drawings. Identical numbers indicate identical or equivalent members in the 
drawings, and explanations of such members will not be repeated. 
[0033] Fig. 1 is a block diagram showing the hardware construction of the 
resolution converting device comprising one embodiment of the present invention. With 
reference to Fig. 1, the resolution converting device includes a controller 100, an image 
input unit 101 that inputs an image of a person's face, an operating unit 102 by which 
the user inputs data and various instructions, a memory 103 that stores the program 
executed by the controller 100 and information necessary for the controller to execute 
the program, an output unit 1 04 that outputs an image, and an external storage device 
105. 

[0034] The controller 100 comprises a central processing unit (CPU), and controls 
the entire resolution converting device. The image input unit 101 comprises an image 



scanner. It has a line-type CCD (Charge Coupled Device) sensor, reads a photograph or 
other representation of a person's face, and outputs two-dimensional face image data. A 
digital camera that can capture the image of a person's face and output a two- 
dimensional image may be used instead. The controller 100 may further comprise an 
input terminal enabling connection to an external image scanner or digital camera. 
[0035] The memory 103 comprises a read-only memory (ROM) in which the 
program executed by the controller 100 is stored, a random access memory (RAM) in 
which variables, etc. necessary for the controller 100 to execute the program are 
temporarily stored, and a hard disk in which various items of data are stored. 
[0036] The output unit 104 comprises a display monitor such as a Braun tube or 
liquid crystal display device on which to display the image input by the image input unit 
101 or the image obtained following image processing such as resolution conversion. A 
printer may also be used together with the display. The operating unit comprises a 
keyboard and a pointing device such as a mouse. 

[0037] The external storage device 105 comprises an optomagnetic disk drive or 
digital video disk drive to read the program stored in the recording medium 1 06 and 
executed by the controller 100, or to read two-dimensional face image data. The data 
regarding the face image obtained through resolution conversion carried out by the 
controller 100 may also be written to the external storage device 105. 
[0038] For the recording medium 106, tape media such as a magnetic tape or 
cassette tape, disk media such as a magnetic disk (flexible disk, hard disk) or an optical 
disk (CD-ROM, MO, MD, DVD), card media such as an IC card (including memory 
card) or an optical card, or media that carry programs in a fixed fashion, such as a 



masked ROM, an EPROM, an EEPROM or a semiconductor memory, i.e., a flash 
memory, may be used. It may also comprise a medium that carries programs in an 
intangible form, such as where a program is downloaded from a network. The external 
storage device handles the recording medium 106 such that information may be read 
from or written to such a recording medium 106. 

[0039] Fig. 2 is a functional block diagram showing the outline of the model 
building feature of the resolution converting device of this embodiment. The resolution 
converting device includes a model building unit 115 that generates a model expressed 
by base vectors sought by performing principal component analysis based on the 
characteristic data of the images input by the image input unit 101, and a hierarchy 
model generating unit 113 that generates models having different resolutions using the 
model generated by the model building unit 115. 

[0040] The model generated by the model building unit 1 1 5 is defined by eigen 
vectors for the bases. The eigen vectors for the bases that comprise the model and the 
eigen vectors for the bases that comprise the models generated by the hierarchy model 
generating unit 1 13 are stored in the memory 103. The face images input by the image 
input unit 101 and used for the generation of a model are stored in the memory 103 
together with the characteristic parameters obtained using that model. 
[0041] The model building unit 115 performs principal component analysis of the 
multiple face images input by the image input unit 101 using the characteristic data 
pertaining to the input images. The characteristic data includes, for example, the 
coordinate and pixel value for each pixel. Therefore, if the resolution is high, the 
quantity of characteristic data increases. The normal orthogonal coordinate system 



sought as a result of the principal component analysis is stored in the memory 103 as a 
model. The image data input by the image input unit 101, the characteristic data that 
corresponds to each set of image data, and the characteristic parameters obtained using 
the model, are stored in the memory 103 in a linked fashion, although this step may be 
unnecessary. 

[0042] Fig. 3 is a flow chart showing the sequence of the model building process 
performed by the resolution converting device of this embodiment. With reference to 
Fig. 3, a standard face image is input by the image input unit 101 (step S01). Here, 
'standard face image 1 comprises a face image used to enable the model building unit 115 
to build a model. Multiple standard face images are available. 
[0043] The standard face image input by the image input unit 101 is then 
normalized (step S02). 'Normalization 1 means matching of the input standard face image 
to a prescribed standard size. Specifically, this is accomplished by matching the distance 
between the eyes of the face image to a prescribed value. Here, in normalization, 
however, the standard face image is reduced only, and is not enlarged. It is preferred that 
all standard face images have the same resolutions. 

[0044] The coordinate and density value of each pixel is input as characteristic data 
regarding the standard image input by the image input unit 101 (step S03). 
[0045] In step S04, it is determined whether or not another standard face image to 
input exists. This determination is made based on the signal input by the user via the 
operating unit 102. Where there is another standard face image to input, the processes of 
the above steps S01 through S03 are repeated with regard to the newly input standard 
face image. Where there are no other face images to input, the process advances to step 
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SOS. By performing the processes of steps SOI through S03 on multiple standard face 
images, characteristic data is input for each of the multiple standard face images. 
[0046] In step SOS, principal component analysis is performed with regard to all 
characteristic data for the standard face images input by the image input unit 101. 
Because principal component analysis is a public-domain statistical technique, 
explanation thereof will be omitted here. When the principal components are extracted 
in step SOS, it is determined in step S06 whether or not the sum of the proportions 
thereof is larger than the threshold value T. Consequently, principal component analysis 
is repeated in step SOS until the sum of the proportions exceeds the threshold value T. 
[0047] Where the sum of the proportions exceeds the threshold value T, the eigen 
vectors for the bases that represent the principal components are stored in the memory 
103 in step S07. A model is built using the eigen vectors for the bases that are stored in t 
the memory 103. 

[0048] In the next step S09, the characteristic parameters of the model regarding the 
standard face images input in step S01 are sought, and the standard face images and the 
characteristic parameters are stored in the memory 103 in a linked fashion. 
[0049] A model is generated from the standard face images, and a face image 
database including the data that associates the standard face images used to generate the 
model and the model's characteristic parameters is created. The standard face images 
may be derived from different persons 1 faces or from the same person's face. Where they 
are derived from the same person, the characteristic parameters representing the same 
person's face are distributed in clusters based on the differences in facial expression. 
[0050] T.F. Cootes et al., "Active Appearance Models" and "In Burkhardt and 



Neumann, editors, Computer Vision - ECCV 98", Vol. II, Frieburg, Germany, 1999 
disclose models thus generated. Using the active appearance models (AAM) proposed 
by Cootes et al., the characteristic parameters of the input two-dimensional image may 
be obtained by projecting the input two-dimensional image onto a model. More 
specifically, using the input image and any given model parameter, the difference from 
the estimated model generated using the model is sought. The characteristic parameters 
of the image input are calculated at high speed through the learning in advance of the 
relationship between the thus sought difference and the change amount to eliminate the 
difference of the model parameters. 

[0051] The eigen vector for each base that comprises the generated model can be 
undergone resolution conversion in the same manner that the input image is subjected to 
resolution conversion. The resolution-converted eigen vector for each base that 
comprises a model are used as a low-order model with a lower resolution. This is called 
model hierarchization. 

[0052] In image hierarchization, generally, the image is subjected to Gaussian 
filtering, and is reduced by eliminating alternating pixels. This process is repeated 
several times, and reduced images in multiple hierarchical levels are created. The 
reduced images thus generated and comprising multiple hierarchical levels are together 
called a Gaussian pyramid. 

[0053] Fig. 4 shows a Gaussian pyramid in a summary fashion. With reference to 
Fig. 4, the image resolution declines in the order from level 0 to level 3. 
[0054] Model hierarchization, in contrast, may be achieved in the following manner. 
The relationship involving the bases obtained through principal component analysis, the 
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characteristic parameters and the image may be expressed by the equation (1). In this 
case, the eigen vector for the base may be subjected to resolution conversion as in the 
case of image hierarchization. 

[0055] In the same manner as in image hierarchization, the eigen vectors 0 through 
n that comprise the normal orthogonal bases are subjected to filtering via a Gaussian 
filter, some of the eigen vector components are eliminated, and hierarchized normal 
orthogonal bases are created. In this case, while there is a reduction in the eigen vector 
components, the number of eigen vectors itself does not change, and therefore, common 
characteristic parameters may be used for each hierarchical level. Therefore, the 
common characteristic parameters that were sought based on model hierarchization may 
be used for each model. If an image is generated using the same characteristic 
parameters and based on each model in the hierarchy, the generated images comprise a 
Gaussian pyramid that is obtained through the image hierarchization. 
[0056] The equation (2) shows the relationship among the bases of a model 
generated through the hierarchization of the model expressed by the equation (1) and 
having a one-quarter resolution, the characteristic vector and the image vector. 
[0057] As is clear from the comparison between the equations (2) and (1), there is 
no change in the number of dimensions in the characteristic vector. In contrast, the 
number of dimensions of the eigen vectors for the bases of each model is reduced to 
one-quarter of those present before the resolution conversion. At the same time, there is 
no change in the number of eigen vectors. The number of dimensions of the image 
vector is one-quarter of the number of dimensions prior to conversion. 
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[0058] Fig. 5 is a block diagram showing in a summary fashion the resolution 
conversion feature of the resolution converting device of this embodiment. The 
resolution converting device includes a face area extracting unit 151 that extracts the 
face area, which shows the face, from the image input by the image input unit 101, an 
image enlarging unit 153 that (i) based on linear or tertiary interpolation, interpolates the 
background area, in which the background other than the face is shown, and (ii) enlarges 
the image, characteristic parameter calculating unit 155 that (i) projects the face area 
image extracted by the face area extracting unit 151 onto a model stored in the memory 
103, and (ii) calculates the characteristic parameters, an image generating unit 157 that 
generates an image by applying the characteristic parameters sought by the characteristic 
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parameter calculating unit 155 to a different model stored in the memory 103, an image 
synthesizing unit 1 59 that synthesizes the image generated by the image generating unit 
157 and the image enlarged by the image enlarging unit 153, and an output unit 104 that 
outputs the synthesized image. 

[0059] The memory 1 03 stores the model generated through the above model 
building process, and the multiple resolution-converted models obtained through the 
model hierarchization are stored as hierarchized models. 

[0060] The face area extracting unit 151 extracts from the image input by the image 

□ 

input unit 101 the face area, in which the face is shown, transmits the face area image to 

*£■ 

03 the characteristic parameter calculating unit 1 55, and transmits the background image, 

r*J; which does not include the face area, to the image enlarging unit 153. 

^ [0061] The characteristic parameter calculating: unit 155 extracts a model that 

_M» corresponds to the resolution of the face area image received from the face area 

111 

extracting unit 151 from among the hierarchized models stored in the memory 103, and 
projects the face area image onto the extracted model. This projection is accomplished 
via the characteristic parameter calculation described above. In other words, the 
difference between the face area image and the model image generated based on a 
model using any given characteristic parameter is sought, the change amount for the 
characteristic parameters is sought based on the difference, and the characteristic 
parameters that correspond to the face area image are calculated based on the thus 
sought change amount. 

[0062] The image enlarging unit 1 53 performs linear or tertiary interpolation of the 
background image received from the face area extracting unit 1 5 1 in order to enlarge the 
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background image. 

[0063] The image generating unit 157 extracts a model expressed using the desired 
resolution from among the hierarchized models stored in the memory 103, and generates 
an image by applying to the extracted model the characteristic parameters calculated by 
the characteristic parameter calculating unit 155. The resolution of the face image 
generated by the image generating unit 157 and the background image enlarged by the 
image enlarging unit 1 53 have the same resolution. — ^ 
[0064] The image synthesizing unit 159 synthesizes the face area image generated \ 
by the image generating unit 157 and the background image enlarged by the image 
enlarging unit 153. The synthesized image is output to the output unit 104 and is 
displayed on the display monitor or printed by the printer. 

[0065] Fig. 6 shows in a summary fashion the resolution conversion process 
performed by the resolution converting device of this embodiment. With reference to 
Fig. 6, the hierarchized models include a model 161, a model 162 having a higher 
resolution than the model 161, and a model 163 having a higher resolution than the 
model 162. The model 161 has a resolution level 0, the model 162 has a resolution level 
1, and the model 163 has a resolution level 2. The resolution of the model 163 is four 
times as high as that of the model 162, and the resolution of the model 162 is four times 
as high as that of the model 161 . In this case, the model 1 63 is generated via the model 
building process shown in Fig. 3. Through resolution conversion of the model 163, 
model hierarchization is accomplished, whereby the models 162 and 161 are obtained. 
[0066] The model 161 that corresponds to the resolution of the input image 165 is 
extracted, and the input image 165 is projected onto the model 161. Based on this 



- 16- 



projection, the characteristic parameters 167 are sought. By generating an image based 
on the model 163 and using the characteristic parameters, the image 169 is generated. 
As a result, the resolution of the image 169 is 16 times higher than that of the input 
image 165. The resolution of the image generated based on the model 162 and using the 
characteristic parameters 167 is four times higher than that of the input image 165. 
[0067] Fig. 7 is a flow chart showing the sequence of the resolution conversion 
process performed by the resolution converting device of this embodiment. With 
reference to Fig. 7, in the resolution converting device, an image is input by the image 
input unit 101 (step SI 1). The image of the face area is then extracted from the input 
image by the face area extracting unit 151 (step SI 2). 

[0068] The standard model size that corresponds to the face area image is 
determined (step SI 3). Model size 1 means the size of a model that corresponds to the 
resolution of the face area image, which is selected from among the hierarchized models 
stored in the memory 103. Hierarchized models are hierarchized based on their 
resolutions, as described above. The resolution of each model of the hierarchy 
corresponds to the resolution of an image. Therefore, in step SI 3, based on the 
resolution of the face area image, a model that corresponds to that resolution is selected 
from among the hierarchized models. The model that corresponds to the resolution of 
the face area image comprises a model that has a resolution that is smaller than the 
resolution of the face area image but is closest to the face area image resolution. 
[0069] In step SI 4, the face area image is filtered. Here, Gaussian filtering is 
performed in order to achieve harmonization with the model. 

[0070] Characteristic parameters are then calculated by the characteristic parameter 
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calculating unit 155, which projects the face area image onto the model selected in step 
S13. 

[0071] In step SI 6, a higher-order model than the model selected in step S13 is 
extracted by the image generating unit 1 57, whereupon an image is generated, higher- 
order model* means a model that has a higher resolution. The higher-order model 
extracted in this step is determined based on the magnification by which to enlarge the 
face area image. In other words, where the resolution is to be increased twice, a model 
having twice the resolution is extracted as a higher-order model. An image is then 

O 

*0 generated by applying the characteristic parameters calculated in step S 1 5 to the higher- 

03 order model extracted. 

Hi! [0072] The background image, which does not include the face area, is subjected to 

till 

t il 

~ linear or tertiary interpolation (or wavelet) by the image enlarging unit 1 53 to convert 

M- 

H> the resolution (step S 1 7). The resolution conversion in this step is based on the 

! r* magnification by which the face area image is enlarged, as in the case of the extraction 

□ 

^ of a higher-order model in step SI 6. Therefore, the face area image and the background 

image have the same resolution. 

[0073] In step SI 8, the face area image generated in step SI 6 and the background 
image, which has undergone resolution conversion in step SI 7, are synthesized and 
output to the output unit 104. 

[0074] In this embodiment, resolution conversion pertaining to a face image was 
explained, but the present invention is not limited to this implementation, and may be 
applied to all types of objects. 

[0075] As explained above, because the resolution converting device of this 
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embodiment enlarges the input image using hierarchized models, a sharp image without 
blurriness may be generated. In addition, through the model hierarchization, the same 
characteristic parameters may be commonly used throughout all models of the hierarchy. 
Therefore, simply by obtaining the characteristic parameters of one model for one 
image, the obtained characteristic parameters may be commonly used with regard to 
other models in the hierarchy, so that the resolution of an image may be easily 
converted. 

[0076] The processing performed by the resolution converting device of this 
embodiment is described in terms of a program, and the same feature as the resolution 
conversion process described above may be attained by having a computer carry out the 
program. 

[0077] The embodiment disclosed herein is merely an example in terms of all 
aspects, and is not restrictive. The scope of the present invention is indicated by the 
claims, not by the explanation provided above, and all modifications equivalent to and 
within the scope of the claims are included. 
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